Reorganize development docs #1200
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03%
- Complexity 3550 3609 +59
============================================
Files 344 344
Lines 8762 8946 +184
Branches 554 569 +15
============================================
+ Hits 8615 8799 +184
Misses 142 142
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
dai-chen
force-pushed
the
refactor-dev-docs-rebased
branch
from
e31b2c0
to
3ad530d
Compare
dai-chen
changed the title
Signed-off-by: Chen Dai <[email protected]>
dai-chen
force-pushed
the
refactor-dev-docs-rebased
branch
from
3ad530d
to
7b038c4
Compare
Signed-off-by: Chen Dai <[email protected]>
Signed-off-by: Chen Dai <[email protected]>
Signed-off-by: Chen Dai <[email protected]>
dai-chen
changed the title
Signed-off-by: Chen Dai <[email protected]>
dai-chen
changed the title
| + **CLI** | ||
| + **JDBC Driver** | ||
| + **ODBC Driver** | ||
| + **Query Workbench** |
There was a problem hiding this comment.
Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?
There was a problem hiding this comment.
yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.
|
|
||
| # OpenSearch SQL/PPL Engine Development Manual | ||
|
|
||
| ## Introduction |
There was a problem hiding this comment.
can we also have tutorial section for playground like exploring
There was a problem hiding this comment.
sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.
| * *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application. | ||
|
|
||
|
|
||
| ## 3.Tenets |
There was a problem hiding this comment.
| ## 3.Tenets | |
| ## 3. Tenets |
| * Changes to PPL Query Language grammar should be simple and easy to onboard. | ||
| * Component design should be extensible for supporting new data stores. | ||
|
|
||
| ## 4.Out of Scope for the Design. |
There was a problem hiding this comment.
| ## 4.Out of Scope for the Design. | |
| ## 4. Out of Scope for the Design |
|
|
||
|
|
||
|
|
|
|
||
|
|
||
|
|
||
| ### 6.1 Data source representation. |
There was a problem hiding this comment.
| ### 6.1 Data source representation. | |
| ### 6.1 Data source representation |
| * *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different. | ||
|
|
||
| Example Prometheus Catalog Definition | ||
| ``` |
There was a problem hiding this comment.
| ``` | |
| ```json |
|
|
||
| * JSON Format |
There was a problem hiding this comment.
| * JSON Format |
- JSON Format
| [ | ||
| {"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)}, | ||
| {"age": ExprNullValue(), "account_number": ExprIntegerValue(2)}, | ||
| {"account_number": ExprIntegerValue(3)} | ||
| ] |
There was a problem hiding this comment.
| [ | |
| {"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)}, | |
| {"age": ExprNullValue(), "account_number": ExprIntegerValue(2)}, | |
| {"account_number": ExprIntegerValue(3)} | |
| ] | |
| ```json | |
| [ | |
| {"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)}, | |
| {"age": ExprNullValue(), "account_number": ExprIntegerValue(2)}, | |
| {"account_number": ExprIntegerValue(3)} | |
| ] |
``
| ##### Include NULL and MISSING value in the QueryResult (Issue 1, 2) | ||
| The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows. | ||
|
|
||
| ``` |
There was a problem hiding this comment.
| ``` | |
| ```json |
|
|
||
| An additional list of Schema is also required to when protocol is JDBC. | ||
|
|
||
| ``` |
There was a problem hiding this comment.
| ``` | |
| ```json |
| 1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch. | ||
| 2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below. | ||
|
|
||
| ``` |
There was a problem hiding this comment.
| ``` | |
| ```java |
|
This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future. |
Signed-off-by: Chen Dai [email protected]
Description
This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in
docs/dev/folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.Please see
https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md-> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.mdIssues Resolved
#1219
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.